Add CpcUnion.update(MemorySegment) - #759
Open
davecromberge wants to merge 1 commit into
Open
Conversation
Merging a stored sketch had to uncompress it into a CpcSketch first, build that sketch's pair table, and then walk the table to OR its coupons into the union's bit matrix. Where the union already holds a bit matrix the coupons can be decoded straight into it, which skips the pair table and the second walk. Sparse, Hybrid and Pinned images decode directly, reusing the union's existing orWindowIntoMatrix. Sliding partly inverts its logic, so a coupon can be signalled by the absence of a pair in the surprises table; those, and unions still holding a sparse accumulator, uncompress a sketch as before. uncompressTheWindow now returns the window instead of assigning it into a target sketch, matching uncompressTheSurprisingValues beside it, so both decode primitives can serve either caller. Measured with a new characterization profile that merges 32 stored sketches at lgK=12: roughly a third less time per sketch across the range that decodes directly, and unchanged where it falls back. The profile's fallback arm runs identical code in both configurations, so its spread bounds the noise at a few percent.
davecromberge
added a commit
to permutive-engineering/datasketches-java
that referenced
this pull request
Sep 3, 2026
Merging a stored sketch had to uncompress it into a CpcSketch first, build that sketch's pair table, and then walk the table to OR its coupons into the union's bit matrix. Where the union already holds a bit matrix the coupons can be decoded straight into it, which skips the pair table and the second walk. Sparse, Hybrid and Pinned images decode directly, reusing the union's existing orWindowIntoMatrix. Sliding partly inverts its logic, so a coupon can be signalled by the absence of a pair in the surprises table; those, and unions still holding a sparse accumulator, uncompress a sketch as before. uncompressTheWindow now returns the window instead of assigning it into a target sketch, matching uncompressTheSurprisingValues beside it, so both decode primitives can serve either caller. Measured with a new characterization profile that merges 32 stored sketches at lgK=12: roughly a third less time per sketch across the range that decodes directly, and unchanged where it falls back. The profile's fallback arm runs identical code in both configurations, so its spread bounds the noise at a few percent. Cherry-picked from apache#759.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
CpcUniononly accepts aCpcSketch, so merging a stored sketch means deserializing it first:That builds an entire sketch — sliding window, pair table, HIP registers — which the union then walks once and discards. This adds an overload that skips it:
Where the union already holds a bit matrix, the image's coupons are decoded straight into it. Sparse, Hybrid and Pinned images decode directly. Sliding partly inverts its logic — a coupon can be signalled by the absence of a pair — so it still goes through a sketch. Either way the resulting union is byte-identical.
Use case
Unioning columns of serialized sketches, which is what query engines and offline rollup jobs mostly do. In Pinot this path is taken by
distinctCountRawCpcSketch, by segment rollup, and by star-tree index construction; all three heapify every input today.How much it helps depends on the flavor of the stored sketches, which follows their cardinality — see below.
Measured
New characterization profile
CpcUnionDeserializeSpeedProfile— 32 stored sketches merged per trial at lgK=12 — median of 4 interleaved runs of each entry point against the same jar. It sweeps 29 unique counts from 256 to 4.2M, grouped here by the flavor the stored images take:The saving tracks how many of an image's coupons live in the surprises table rather than the sliding window, because both paths decode the window identically:
The gain is largest for columns of many small sketches, and falls away for columns of few large
ones, where the union pays for a window decode either way.